1. Field of the Invention
The present invention relates to a method for estimating the placement information of markers disposed within a 3-D space.
2. Description of the Related Art
In recent years, studies regarding MR (Mixed Reality) technology have been actively pursued. The MR technology is used for seamlessly integrating real space with virtual space created by a computer. Of the MR technology, AR (Augmented Reality; also referred to as enhanced reality) technology which superimposes virtual space over real space has particularly attracted attention.
An image display device on which the AR technology displays an image is realized by a video see-through method which displays a synthesized image obtained by superimposing a computer-generated image concerned with virtual space (a virtual object, character information, or the like, drawn with computer graphics) rendered according to the position and orientation of a later-described imaging device over an image of real space photographed using an imaging device such as a video camera, or by an optical see-through method which displays an image of virtual space rendered according to an observer's viewing position and orientation on an optical-see-through-type display mounted on the observer's head.
There are expectations for application of the AR technology to various fields such as surgical aids which superimposes the state within a patient's body over the surface of the body, an architectural simulation which superimposes a virtual building over an image of vacant land, assembly aids which superimposes assembly procedures and wiring for assembly of machines or other devices, and the like.
The most important problem to be solved in the AR technology is how to accurately perform the registration between real space and virtual space, and heretofore, many methods have been attempted. The registration problem in the AR technology corresponds to a problem of obtaining the position and orientation of an imaging device in a scene (i.e., in the reference coordinate system) in the case of the video see-through method. Similarly, in the case of the optical see-through method, the registration problem corresponds to a problem for obtaining the position and orientation of an observer or a display device in the scene.
A commonly-employed method for solving the former problem is to dispose artificial markers or make natural characteristic points markers in the scene, based on the correspondence between the projected positions of the markers within an image photographed by an imaging device and the positions in the reference coordinate system of the marker so as to obtain the position and orientation of the imaging device in the reference coordinate system. Also, a commonly-employed method for solving the latter problem is to mount the imaging device on a target to be measured (e.g., an observer's head or display), with the position and orientation of the imaging device being obtained in the same way as with the former method, and the position and orientation of the target to be measured is obtained based thereupon.
Description will be made regarding a conventional example of a position-and-orientation measuring device for measuring the position and orientation of a later-described imaging device by correlating the 2-D coordinates of a marker to be detected from an image photographed by an imaging device and the 3-D position of the marker in the reference coordinate system with reference to FIG. 1. As illustrated in FIG. 1, a position-orientation measuring device 100 in the present conventional example includes a marker detecting unit 110 and position-orientation calculating unit 120, and is connected to an imaging device 130.
Also, K markers Qk (k=1, 2, and so on through K) of which the positions in the reference coordinate system are known are disposed in real space as the markers for obtaining the position and orientation of the imaging device 130. The example in FIG. 1 illustrates a situation in which four markers Q1, Q2, Q3, and Q4 are disposed. Of these, three markers Q1, Q3, and Q4 are inside the field of view of the imaging device 130 and one marker Q2 is outside the field of view of the imaging device 130.
The markers Qk can be any shape, such as a circular marker having a different color from other markers, or the like, as long as the projected position of a marker within a photographed image can be detected, and also the marker can be identified. For example, natural characteristic points within 3-D space may be employed, and such points may be detected within a photographed image using template matching. An image output from the imaging device 130 is input to the position-orientation measuring device 100. The marker detecting unit 110 inputs an image by the imaging device 130, and detects the image coordinates of the markers Qk photographed on the image. For example, in the event that each of the markers Qk is made up of a circular marker each having a different color, the marker detecting unit 110 detects a region corresponding to each marker color from on the input image, and takes the barycentric position as the detected coordinates of the marker.
Further, the marker detecting unit 110 outputs the image coordinates uMkn of each detected marker Qkn and the identifier kn thereof to the position-orientation calculating unit 120. Here, n (=1, 2, and so on through N) is a symbol representing the serial number of the detected markers, and N represents the total number of the detected markers. For example, in the case of FIG. 1, N=3, so the identifiers k1=1, k2=3, k3=4 and the image coordinates uMk1, uMk2, and uMk3 corresponding to these are output.
The position-orientation calculating unit 120 calculates the position and orientation of the imaging device 130 based on the correlation between the image coordinates uMkn of each detected marker Qkn and the position in the reference coordinate system of the marker Qkn, which is held as known information beforehand. A method for calculating the position and orientation of an imaging device based on a pair of the 3-D coordinates of a marker and image coordinates has been proposed in the field of photogrammetry as of old (for example, see R. M. Haralick, C. Lee, K. Ottenberg, and M. Nolle: “Review and analysis of solutions of the three point perspective pose estimation problem”, Int'l. J. Computer Vision, vol. 13, no. 3, pp. 331-356, 1994 and M. A. Fischler and R. C. Bolles: “Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography”, Comm. ACM, vol. 24, no. 6, pp. 381-395, 1981). The position-orientation calculating unit 120 calculates the position and orientation of the imaging device 130 using the method described in R. M. Haralick, C. Lee, K. Ottenberg, and M. Nolle: “Review and analysis of solutions of the three point perspective pose estimation problem”, Int'l. J. Computer Vision, vol. 13, no. 3, pp. 331-356, 1994, for example.
Note that description has been made regarding the case of employing markers (hereinafter, referred to as “point markers”) as multiple points within 3-D space, but a calculation method for calculating the position and orientation of an imaging device using square-shaped markers (hereinafter, referred to as “square markers”) having a known size has been proposed as disclosed in J. Rekimoto: “Configuration method of augmented reality using 2-D matrix code”, Interactive System and Software IV, Kindai Kagakusha, 1996 and Kato, M. Billinghurst, Asano and Tachibana: “Augmented reality based on marker tracing and calibration thereof”, Japan Virtual Reality Academic Journal, vol. 4, no. 4, pp. 607-616, 1999, for example. A calculation method of the position and orientation of an imaging device using combination of square markers and point markers has been proposed, as disclosed in H. Kato, M. Billinghurst, I. Poupyrev, K. Imamoto and K. Tachibana: “Virtual object manipulation on a table-top AR environment”, Proc. ISAR2000, pp. 111-119, 2000, for example. With this calculation method, point markers have an advantage wherein point markers can set even in a narrow place, square markers have advantages wherein identification is easy, and the position and orientation of the imaging device can be obtained from only one marker since one marker includes sufficient information, thus utilizing these two types of markers in a complementary manner.
According to the aforementioned methods, based on an image photographed by an imaging device, the position and orientation of the imaging device has been acquired since the past.
On the other hand, an arrangement has been made in which a 6-degree-of-freedom position and orientation sensor such as a magnetic sensor, ultrasonic sensor, or the like is attached to an imaging device serving as a target to be measured, and the position and orientation the imaging device is measured by concomitant use with marker detection by image processing as described above, as disclosed in Japanese Patent Laid-Open No. 11-084307, Japanese Patent Laid-Open No. 2000-041173, and A. State, G. Hirota, D. T. Chen, W. F. Garrett and M. A. Livingston: “Superior augmented reality registration by integrating landmark tracking and magnetic tracking”, Proc. SIGGRAPH'96, pp. 429-438, 1996. The accuracy of a sensor output changes depending on a measuring range, but can be obtained robustly, so a method using both sensor and image processing can improve robustness as compared with a method using image processing alone.
With a registration method using markers, the position in the reference coordinate system in the case of point markers and the position and orientation in the reference coordinate system in the case of square markers needs to be known for obtaining the position and orientation in the reference coordinate system of an imaging device serving as a target to be measured. In the case of a square marker, the square marker itself is often taken as the reference of the coordinate system without separately providing the reference coordinate system, but in the case of employing multiple square markers, the mutual position and orientation relations need to be known, and accordingly, there is no difference in that the reference coordinate system needs to be employed.
The position and orientation of a marker may be measured by hand using a measuring tape, ruler, or protractor, or by a surveying instrument, but measurement techniques utilizing images have been performed to improve accuracy and save time. The position of a point marker can be measured by a method called the bundle adjustment method. The bundle adjustment method is a method in which a great number of point markers are photographed by an imaging device, the position and orientation of the imaging device taking each image and the positions of point markers are obtained by repeated calculation so that the error between the projected positions where the markers are actually observed on the image, and the projected positions to be calculated from the position and orientation of the imaging device, and the positions of the markers, can be minimized.
Also, a method for measuring the position and orientation of multiple square markers disposed within 3-D space has been disclosed in G. Baratoff, A. Neubeck and H. Regenbrecht: “Interactive multi-marker calibration for augmented reality applications”, Proc. ISMAR2002, pp. 107-116, 2002. With G. Baratoff, A. Neubeck and H. Regenbrecht: “Interactive multi-marker calibration for augmented reality applications”, Proc. ISMAR2002, pp. 107-116, 2002, the position and orientation of an imaging device taking each image, and the position and orientation of each square marker are obtained using a method in which a great number of images of multiple square markers disposed within 3-D space are photographed, and repeated calculation is performed so that projection error can be minimized.
With the aforementioned conventional methods for measuring the position and orientation of a marker, the position of a point marker, and the position and orientation of a “planar” marker such as a square marker cannot be obtained simultaneously. More specifically, the placement information of multiple types of markers, of which the placement information has different amounts of information, such as a marker (point marker, for example) having placement information, i.e., the position of one point within 3-D space, a marker (plane marker, for example) having placement information, i.e., the position and orientation within 3-D space, and the like, cannot be obtained simultaneously. Consequently, the placement information of markers such as the position of point markers, the positions and orientations of plane markers, and the like need to be obtained independently for each type of markers. Accordingly, even if different types of marker (point marker and plane marker, for example) had been photographed on the same image, the placement information of each type of marker, and the position and orientation of an imaging device so as to minimize the projection error cannot be obtained, so the placement information of the markers cannot be obtained accurately.